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ABSTRACT 

A study was conducted to determine how best to use 
APS Reading Test scores for making English placement recommendations 
for students at College of the Canyons (COC) in Santa Clarita, 
California. The primary method used to establish cut scores involves 
comparing success rates for contrasting groups over a range of 
scores. The contrasting groups are defined as successful students who 
earned grades of A, B, C and CR; and unsuccessful students earning 
grades of D, F, NC or withdrawal. The study population included 213 
students from spring 1993, fall 1993 or spring 1994 enrolled in 
English 010 (N=15), English 034 (N-88) , English 080 (N=*70) , or 
English 101 (N=40) . The students were first-time college students 
that had taken the APS Reading Test just prior to enrolling in 
English. Scatter diagrams were prepared to illustrate the 
"relationship between test scores and success in each target class. 
The scattergrams provide an initial basis for establishing trial cut 
scores to minimize placement errors* The findings of the study 
validated the use of the APS Reading Test to support placement 
decisions for students entering English 034, 080 and 101. Use of the 
test together with proposed placement rules could increase the rate 
of correct placement over the base rate by 9.1% for English 034, and 
2.6% for English 101. (KP) 
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VALIDATING PLACEMENT RULES 
FOR THE APS READING TEST 

The report entitled Predictive Validity Study of the APS Reading Test (July 1994) showed 
the adequate overall relationship between test scores and course grades for the APS Reading Test, 
supporting the use of the test scores as predictors of student performance at College of the 
Canyons. 

The next issue is a very practical one - how best to use the tests for making placement 
recommendations. This involves identifying cut scores. At what point in the continuum of test 
scores do we recommend placement of students in a given course? For example, are students 
who receive scores of 25 to 35 on the APS Reading Test fully prepared to undertake English 101 
- English Composition and Literature? Should the score range be broader or narrower? No 
method of identifying cut scores is flawless. With typical validities of .35, only about 12 percent 
of the variance in student performance is associated with the test scores. A great many factors 
not tapped by the tests contribute to course grades. The goal of the process is to improve 
prediction of success. 

METHOD 

The primary method used lor establishing cut scores involves comparing success rates for 
contrasting groups over a range of scores. More specifically, the contrasting groups are defined 
as "successful students" or those who earned grades of A, B, C and CR, whereas persons with 
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grades of D, F, NC or W (withdrawal) are categorized in the "unsuccessful" group. Given this 
distinction, it is then possible to determine the proportion of students with specific scores, and 
hence for score ranges, that were successful. Cut scores are then evaluated in terms of their 
ability to "predict" success (i.e. maximize the difference between success rates for persons scoring 
above a value and those scoring below it), 

Sulye£lS. The research participants were 213 students from spring 1993, fall 1993 or spring 
1994. The students enrolled in English 010 (N=15), English 034 (N=88)> English 080 (N=70), 
or English 101 (N=40). The students were first-time college students and had taken the APS 
Reading Test just prior to enrolling in the English course. Participating students were placed in 
English courses using existing placement procedures, including the use of placement test scores. 
End-of-course grades were used as the student performance criterion. 

RESULTS 

Placement Rules . The scatter diagrams presented in Figures 1 through 3 show the relationship 
between test scores and success in each target class. These provide an initial basis for 
establishing trial cut scores to minimize placement errors. On the scatter plots, a number 
represents one or more coincident data points while letters designate two-digit numbers, i.e. 
A=10, B=ll, C=12 etc. Withdrawals were treated as unsuccessful course outcomes in this 
analysis of dichotomized grades. 
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Figure 1. 

English 034 Success by APS Reading Test 



College of the Canyons 

Scatter Diagram 09/20/94 **APRS* 

Data: Read SF93 - S94 09:42:33 Page 1 

Test: READING 02: APS READING A 
Subgroup: TOTAL 
N=88, R=0.19 
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Figure 2. 

English 080 Success by APS Reading Test 

College of the Canyons 

Scatter Diagram 09.42.33 p tge 2 

Data: Read SF93 - S94 

Test READING 02: APS READING A 

Subgroup: TOTAL 

N=70, R=0.H 
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Figure 3. 

English 101 Success by APS Reading Test 
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Scatter Diagram 

Data: Read SF93 - S94 

Course: English 101 

Test: READING 02: APS READING A 

Subgroup: TOTAL 

N=39. R=-0.07 
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The scatterplots were partitioned using a variety of possible cut scores. The partitions 
produced placement classification tables for each trial cut score. Each table partitions ihe data 
from the scatterplot into four quadrants. The upper right and lower left quadrants represent 
correct placements. The other two quadrants represent placement errors. 

The analysis showed that there were more than one set of cut scores for each course that 
were optimum in placement accuracy. The recommended cut scores shown in Table 1 were set 
in the lower part of the optimum range for each course. 



Table 1. 

Recommended APS Reading Test Cut Scores by Course 



Test 


Score Range 


Recommended 
Placement 


% of Study Sample 
in Score Range 


N 


Writing 


0 - 11 


English 010 


15.1% 


32 




12 - 15 


English 034 


33.0% 


70 




16 - 21 


English 080 


22.7% 


48 




22 - 35 


English 101 


29.2% 


62 
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Current APS Reading Test Cut Scores By Course 


lest 


Score Range 


Recommended 
Placement 


% of Study Sample 
in Score Range 


N 


Writing 


0 - 9 


English 010 


6.1% 


13 




10 - 15 


English 034 


42.0% 


89 




16 - 24 


English 080 


36.8% 


78 




25 - 35 


English 101 


15.1% 


32 



Placement Accuracy. Tables 2 through 4 are placement classification tables that show the 
relationships between admission status (that would have resulted from use o£ the recommended 
placement rules) and actual student performance observed in the course. For this analysis, 
students were considered eligible for the course they selected if their score exceeded the cut 
score, even if that score would have qualified them for admission to a higher level course. 



Course Outcome 



Placement 
Recommendation 



Not 

Eligible 



Successful 


A 


B 


Not 


C 


D 


Successful 







Eligible 



Correct classifications are represented in the upper right (B) and lower-left (C) cells of the table, 
the sum of these cells represents the total of correct placement. The percentage of correct 

*i0 



Matriculation 

classifications is shown as a footnote to each table. In addition, the footnotes show the net gain 
in placement accuracy that would result from use of the recommended cut score and the 
percentage of students eligible (selection ratio) under this placement rule, A placement rule or 
cut score is effective when the gain in placement accuracy is significant. This gain is defined 
as the difference between the correct placements and the base rate, where the base rate is the 
number of students who would have succeeded in the course if enrollment were open (A+B). 
Following is an explanation of the above in formula notation: 

Base Rate = (A + B)/(A + B + C + D) 

Correct Placements = (B + C)/(A + B + C + D) 

Gain in Placement Accuracy = Correct Placements - Base Rate 

Selection Ratio = (B + D)/(A + B + C + D) 
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Table 2. 
Percentage of Students by 
English 034 Success and Admission Status 



CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 

Database: Read SF93 - S94 (Filter: NOSE J 
Cc;rse: ENGL 034 

Test: REAPING 02: APS READING A 

Si:bcro-;p: TOTAL IN = 88 J 



N'OT ELIG ELIGIBLE 
< 12 >= 12 
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Table 3. 
Percentage of Students by 
English 080 Success and Admission Status 



C. ASSIF : CATION TABLES FDR OPTIMAL CL'T SCORES 
Database- Read SF?3 - S94 (Filter: NONE J 
Ccurse: ENGL CSC 

Test: READING 02: APS READING A 
Subgroup: TOTAL (N = 70) 





NOT ELIG 


ELIGIBLE 




< 16 


>= 16 






7 


48 


SL'CCESF 


10. 


C% 


68.6% 


A.B.C.CR 




7% 


r 37.3% 




c 8 7. 




c 77.4% 
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14 


NTT Sl'CC 


1 . 


4% 
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D • F , NO , W 


r 6. 


7% 


r 93.3% 




c ) I . 


5% 


c 22.6% 
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62 




1 1 . 
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88.6% 




c 100. 


0% 


c 100.0% 
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15 
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70 
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Correct Place.7ier.ts: 70.0% 
Base Rate (s-iccess): 78.6% 
Gain in Accuracy: ~8.6% 



Selection Ratio: 8?. 6% 

Diff in Succ Ratio: -If, IX 
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Table 4. 
Percentage of Students by 
English 101 Success and Admission Status 



CLASSIFICATION TABLES FOR OPTIMAL CVT SCORES 
Database: Read SF93 - S94 (Filter: NONE: 
Course : ENGL 101 

Test: READING 02: APS READING A 
Subgroup: TOTAL ( N s 39) 
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IG 
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< 22 
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24 
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6% 
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Correct Placements : 66.7% 
Base Rate (success): 64.1% 
Gain in Accuracy: 2.6% 



Selection Ratio: 92.3% 
Dxff in Succ Ratic: 33.3% 



The following line graphs provide a visual depiction of three of the major factors 
considered when selecting the appropriate cut scores: (1) percentage of correct placements; (2) 
percentage of false positives - or those who were eligible but who received an unsuccessful final 
grade; and (3) percentage of false negatives - or those who were ineligible but who received a 
successful final grade. One common error in selecting cut scores is to only look at the 
percentage of correct placements. By doing so, it encourages cut scores that are too high. 
Figures 4 through 6 show the recommended cut scores for English 034, 080 and 101. 
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Figure 4. 

English 034 Recommended Cut Score of 12 

CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 

Data: Read SF93 - S94 

Course: English 034 

Test: READING 02: APS READING A 

Subgroup: TOTAL 

N=88 
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Figure 5. 

English 080 Recommended Cut Score of 16 

CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 

Data: Read SF93_^ S94 

Course: English 080 

Test: READING 02: APS READING A 

Subgroup: TOTAL 

N=70 




Figure 6. 

English 101 Recommended Cut Score of 22 

CLASSIFICATION TABLES FOR OPTIMAL CUT SCORES 

Date: Read SF93 - S94 

Course: English 101 

Test READING 02: APS READING A 

Subgroup: TOTAL 

N=39 
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Table 5 provides supplemental information in support of the relationship between test 
scores and grades. Specifically, the mean test scores for key criterion (grade) defined groups — 
for students earning an A or B grade, a C or Credit grade, a grade of D, F or No Credit, and 
those receiving Ws — are presented. With the exception of English 101, the mean test scores for 
the successful students is greater than the mean values for students who earned unsuccessful 
grades of D, F or No Credit. The mean of 29,00, for students enrolled in English 101 who 
received final grades of D, F or No Credit, is based on an N of 6, a small number of cases. 
While the pattern is not a perfect linear relationship, these data are consistent with the overall 
linear relationships between test scores and course grades, and also support the reasonableness 
of the recommended cut score levels, 
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Table 5. 

Mean Score for Key Grade-Based Criterion Groups, 
By Course 



Final Grade Earned 

Test/Course A/B C/CR D/F/NC W 



APS Reading Test 



English 010 


* 


11.00 


* 


6.00 


English 034 


13.86 


14.36 


13.00 


13.15 


English 080 


19.40 


19.35 


17.71 


19.13 


English 101 


26.38 


25.50 


29.00 


24.63 



* N is 5 or less. 



Since the study sample represents only a portion of the students who actually took the 
APS Reading Test in the three semesters, we questioned what proportion of the total group (N 
= 2,755) would have fallen into each of the cut score ranges. Table 6 provides the breakdown. 
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Table 6. 

Recommended APS Reading Test Cut Score by Course 
Spring and Fall 1993 



Test 


Score Range 


Recommended 
Placement 


% of 
in Score Range 


N 


Reading 


0 - 11 


English 010 


8.0% 


221 




12- 15 


English 034 


11.4% 


313 




16 - 21 


English 080 


27.8% 


767 




22 - 35 


English 101 


52.8% 


1,454 



A smaller percentage of the total group of test-takers fell into each of the two lower cut 
score ranges than was true of the study sample. Using the recommended cut scores of 22 - 35, 
the majority (52.8%) of all tested students would have been recommended for placement in 
English 101. This compares to only 29.2% of the total group - had the current cut scores been 
used (25 - 35). 

DISCUSSION 

The findings of this study validate the use of the APS Reading Test to support placement 
decisions for students entering English 034, 080 and 101. Use of the test together with the 
proposed placement rules could increase the rate of correct placement over the base rate by 9.1% 
for English 034, and 2.6% for English 101. There was no gain in accuracy for English 080. 

A thorough review of these findings by all involved faculty and Student Services 
personnel is encouraged. 
CUTAPSW.wp 
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